Multi-task Learning with Gradient Guided Policy Specialization

We present a method for efficient learning of control policies for multiple related robotic motor skills. Our approach consists of two stages: joint training and specialization training. During joint training, a single neural network policy is trained to perform multiple tasks. This forces the policy to learn a common representation of the different tasks. Then, during the specialization training stage we selectively split the weights of the policy based on a per-weight metric that measures the disagreement among the multiple tasks. By splitting part of the control policy, it can be further trained to specialize to each task. To update the control policy during learning, we use Proximal Policy Optimization. We evaluate our approach on two continuous control problems in simulation: 1) training three single-legged robots that have considerable difference in shape and size to hop forward and 2) training a 2D biped robot to walk forward and backward. We test our method with different joint training iterations and specialization amounts and compare our method to a random specialization scheme and a standard specialization scheme. Finally, we design a multitask problem where the inter-task similarity can be continuously controlled. We show that our method can help learning multitask problems for different types of robots and with different levels of similarities among the tasks.